<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <meta name="version" content="S5 1.1" />
  <meta name="author" content="Kristoffer Bjarjefür, Luíza Andrade, Roshni Khincha" />
  <title>Data Management</title>
  <style type="text/css">
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
  </style>
  <!-- configuration parameters -->
  <meta name="defaultView" content="slideshow" />
  <meta name="controlVis" content="hidden" />
  <!-- style sheet links -->
  <link rel="stylesheet" href="www/slides.css" type="text/css" media="projection" id="slideProj" />
  <link rel="stylesheet" href="www/outline.css" type="text/css" media="screen" id="outlineStyle" />
  <link rel="stylesheet" href="www/print.css" type="text/css" media="print" id="slidePrint" />
  <link rel="stylesheet" href="www/opera.css" type="text/css" media="projection" id="operaFix" />
  <!-- S5 JS -->
  <script src="www/slides.js" type="text/javascript"></script>
</head>
<body>
<div class="layout">
<div id="controls"></div>
<div id="currentSlide"></div>
<div id="header"></div>
<div id="footer">
  <h1></h1>
  <h2>Data Management</h2>
</div>
</div>
<div class="presentation">
<div class="title-slide slide">
  <h1 class="title">Data Management</h1>
  <h3 class="author">Kristoffer Bjarjefür, Luíza Andrade, Roshni Khincha</h3>
</div>
<div id="you-do-not-have-time-to-not-set-up-a-system" class="slide section level1">
<h1>You do not have time to not set up a system</h1>
<p>Without a system for <span style="color:orange">effective and efficient collaboration</span>, too much of a team’s data work time is spent on other things than answering questions using the data. For example:</p>
<ul>
<li>Searching for files in the project folder</li>
<li>Fixing errors to make someone else’s code run</li>
<li>Searching in old emails for answers to questions</li>
<li>Guessing how the old versions of the file that generated the last report could have looked like</li>
</ul>
</div>
<div id="creating-a-plan-for-collaboration" class="slide section level1">
<h1>Creating a plan for collaboration</h1>
<p>Try to predict at the start of a project what technical tools you will need at any point in the project. It is ok to update if needed, but always better if avoided.</p>
<ul>
<li>How will data be received?</li>
<li>Where will data be processed and analyzed?</li>
<li>Will only regular computers be needed? Or are server/cloud resources needed?</li>
<li>Which programming language will the team use?</li>
<li>How will you track different versions of
<ul>
<li>Code?</li>
<li>Data?</li>
<li>Outputs?</li>
</ul></li>
<li>How will files and folders be named?</li>
</ul>
</div>
<div id="creating-a-plan-for-collaboration-1" class="slide section level1">
<h1>Creating a plan for collaboration</h1>
<p>Standardizing organization of documents and code prevents mistakes and reduces the cost of transitioning across projects and teams. This includes things as simple as</p>
<ul>
<li>How will we name files?</li>
<li>How will we name the folders where the files are stored?</li>
<li>How will we take notes about the work we are doing?</li>
<li>Where will we save the notes about the work we are doing?</li>
</ul>
</div>
<div id="creating-a-plan-for-collaboration-2" class="slide section level1">
<h1>Creating a plan for collaboration</h1>
<ul>
<li>It may seem silly to spend time planning for something that is so intuitive, but it will make a difference as your data work starts to get more and more complex</li>
<li>What seems intuitive for one person is not necessarily intuitive for another</li>
<li>When a team does not have an open discussion about how they will organize files, people make decisions on the fly, and these decisions may not make sense for other team members (or even to the same person in a few months)</li>
</ul>
</div>
<div id="creating-a-plan-for-collaboration-3" class="slide section level1">
<h1>Creating a plan for collaboration</h1>
<ul>
<li>For the purpose of this course, we have created a folder structure and used naming conventions that follow what we usually do in our department</li>
<li>However, you don’t necessarily need to follow the same structure outside of this course</li>
<li>What matters is that everyone in your team know the structure to be used</li>
</ul>
</div>
<div id="a-mental-model-of-research-data-work" class="slide section level1">
<h1>A mental model of research data work</h1>
<p><img src="img/intro.png" style="width:40.0%" /></p>
</div>
<div id="a-mental-model-of-data-work" class="slide section level1">
<h1>A mental model of data work</h1>
<ul>
<li><strong>Acquiring data:</strong> this is when you receive the data that will be used to answer your questions. The format in which the data will be received often times does not depend on your team, but on whoever is sharing the data with you.</li>
<li><strong>Importing data:</strong> this is when you convert the format of the data from however it was shared with you to the format that best suits the statistical software you will be using.</li>
<li><strong>Cleaning data:</strong> this is when you process the data in the statistical software of your choice so it can be easily handled in that software.</li>
<li><strong>Constructing data:</strong> this is when you will create the indicators the will answer your questions.</li>
<li><strong>Analyzing data:</strong> this is when you will create tables and graphs that contain the answers you were looking for.</li>
</ul>
</div>
<div id="a-mental-model-of-data-work-1" class="slide section level1">
<h1>A mental model of data work</h1>
<ul>
<li>The steps that we outlined are a pedagogical abstraction that help us think about what we are doing</li>
<li>We divided data work into these steps by thinking of how the data was changing over the data work process</li>
<li>In practice, many of these steps will happen at the same time</li>
<li>Still, we will organize our work, including our code and our data, following these steps</li>
<li>This is because the sub tasks in these steps have the same data inputs and outputs</li>
</ul>
</div>
<div id="organizing-data-work" class="slide section level1">
<h1>Organizing data work</h1>
<p><img src="img/datawork-folder.png" style="width:60.0%" /></p>
</div>
<div id="organizing-data" class="slide section level1">
<h1>Organizing data</h1>
<p>Data will be divided by <span style="color:orange">source</span>, <span style="color:orange">unit of observation</span> and <span style="color:orange">stage of data work</span></p>
<p><img src="img/data-folder.png" style="width:60.0%" /></p>
</div>
<div id="organizing-data-1" class="slide section level1">
<h1>Organizing data</h1>
<p>Data will be divided by <span style="color:orange">source</span>, <span style="color:orange">unit of observation</span> and <span style="color:orange">stage of data work</span></p>
<ul>
<li>Each <strong>stage of data work</strong> will be represented by one <strong>folder</strong> inside the <code>Data</code> folder</li>
<li>Each <strong>unit of observation</strong> will be represented by one <strong>file</strong> in the folder that corresponds to a given stage</li>
<li>If you receive data from multiple <strong>sources</strong>, each source will have a separate <strong>folder</strong> inside the <code>Raw</code> data folder</li>
</ul>
</div>
<div id="organizing-code" class="slide section level1">
<h1>Organizing code</h1>
<p>Data will be divided by <span style="color:orange">unit of observation</span> and <span style="color:orange">stage of data work</span></p>
<p><img src="img/code-folder.png" style="width:60.0%" /></p>
</div>
<div id="organizing-code-1" class="slide section level1">
<h1>Organizing code</h1>
<p>Data will be divided by <span style="color:orange">unit of observation</span> and <span style="color:orange">stage of data work</span></p>
<ul>
<li>Each <strong>stage of data work</strong> will be represented by one <strong>folder</strong></li>
<li>Each <strong>unit of observation</strong> will be represented by one <strong>file</strong> in the folder that corresponds to a given stage</li>
</ul>
</div>
<div id="organizing-documentation" class="slide section level1">
<h1>Organizing documentation</h1>
<ul>
<li>Inside the <code>Documentation</code> folder, we will keep all the notes that we take during our project</li>
<li>This folder does not have a pre-determined strucutre because its contents depend a lot on the type of work that we are doing</li>
<li>At the very least, however, it will have three sub-folders:
<ul>
<li><strong>Dictionaries</strong>, where we will store files listing the contents of each data table</li>
<li><strong>Codebook</strong>, where we will store files describing the values found on all of our variables</li>
<li><strong>Data quality issues</strong>, where we will record which issues we found in the data while exploring it, and which observations had issues</li>
</ul></li>
<li>We should add to this folder any notes about communications and decisions that we make about our data</li>
</ul>
</div>
<div id="organizing-outputs" class="slide section level1">
<h1>Organizing outputs</h1>
<ul>
<li>Inside the <code>Output</code> folder, we will keep the graphs and figures that answer our questions</li>
<li>This folder also does not have a pre-determined structure</li>
<li>Some teams like to have folder called <code>Tables</code> and <code>Graphs</code></li>
<li>Some teams like to have folder called <code>Raw</code> and <code>Final</code>, where <code>Raw</code> contains tables and graphs, and <code>Final</code> contains final papers and reports</li>
</ul>
</div>
<div id="organizing-data-work-1" class="slide section level1">
<h1>Organizing data work</h1>
<ul>
<li>When we are <span style="color:orange">importing</span> data, we will write code in the <strong><code>Code/Importing</code></strong> folder that takes data in Excel or csv from the <strong><code>Data/Raw</code></strong> folder and saves data in <code>.dta</code> to the <strong><code>Data/Raw</code></strong> folder</li>
<li>When we are <span style="color:orange">cleaning</span> data, we will write code in the <strong><code>Code/Cleaning</code></strong> folder that takes data <code>.dta</code> from the <strong><code>Data/Raw</code></strong> folder and
<ul>
<li>Saves data in <code>.dta</code> to the <strong><code>Data/Clean</code></strong> folder</li>
<li>Save Excel files to the <strong><code>Documentation/Dictionaries</code></strong> folder</li>
<li>Save files to the <strong><code>Documentation/Codebook</code></strong> folder</li>
</ul></li>
</ul>
</div>
<div id="organizing-data-work-2" class="slide section level1">
<h1>Organizing data work</h1>
<ul>
<li>When we are <span style="color:orange">constructing</span> data, we will write code in the <strong><code>Code/Construction</code></strong> folder that takes data <code>.dta</code> from the <strong><code>Data/Clan</code></strong> folder and
<ul>
<li>Saves data in <code>.dta</code> to the <strong><code>Data/Final</code></strong> folder</li>
<li>Save Excel files to the <strong><code>Documentation/Dictionaries</code></strong> folder</li>
<li>Save files to the <strong><code>Documentation/Codebook</code></strong> folder</li>
</ul></li>
<li>When we are <span style="color:orange">analyzing</span> data, we will write code in the <strong><code>Code/Construction</code></strong> folder that takes data <code>.dta</code> from the <strong><code>Data/Clean</code></strong> folder and save graphs and tables to the <strong><code>Output</code></strong> folder</li>
</ul>
</div>
<div id="referencing-files-in-code" class="slide section level1">
<h1>Referencing files in code</h1>
<ul>
<li>As you can see, we will commonly call files from within code over the different stages of data work</li>
<li>This is actually harder than it may seem</li>
<li>Code refers to files by their <strong>file paths</strong>, that is, where they are stored inside your computer</li>
<li>This can be very tricky as
<ul>
<li>People have a hard time understanding how a computer organizes the data inside its storage</li>
<li>Different computers will store the same file in different places</li>
</ul></li>
</ul>
</div>
<div id="referencing-files-in-code-1" class="slide section level1">
<h1>Referencing files in code</h1>
<ul>
<li>When you opened the data we used earlier today, Stata probably printed something like this on your console:</li>
</ul>
<pre><code>use &quot;E:\DataWork\Data\Final\final_process.dta&quot; </code></pre>
</div>
<div id="referencing-files-in-code-2" class="slide section level1">
<h1>Referencing files in code</h1>
<ul>
<li>When you opened the data we used earlier today, Stata probably printed something like this on your console:</li>
</ul>
<pre><code>use &quot;E:\DataWork\Data\Final\final_process.dta&quot; </code></pre>
<p><span style="border-left: solid 5px lightgray;padding-left: 1em;display: block;margin-block-start: 1em;margin-block-end: 1em;margin-inline-start: 40px;margin-inline-end: 40px;font-size:80%">Do you understand what this means?</span></p>
</div>
<div id="referencing-files-in-code-3" class="slide section level1">
<h1>Referencing files in code</h1>
<p><strong>Question:</strong> What would happen if you connected the flash drive to a different port and tried to run the same code?</p>
</div>
<div id="referencing-files-in-code-4" class="slide section level1">
<h1>Referencing files in code</h1>
<p><strong>Question:</strong> What would happen if you connected the flash drive to a different port and tried to run the same code?</p>
<p><span style="border-left: solid 5px lightgray;padding-left: 1em;display: block;margin-block-start: 1em;margin-block-end: 1em;margin-inline-start: 40px;margin-inline-end: 40px;font-size:80%">Stata would tell you that it cannot find that file. If this happens for the same code in the same computer, can you imagine how difficult it can be to work on different computers?</span></p>
</div>
<div id="referencing-files-in-stata" class="slide section level1">
<h1>Referencing files in Stata</h1>
<ul>
<li>Fortunately, Stata has made our lives easier by creating a <strong>Project Management</strong> tool</li>
<li>This tool lets you refer to a set of files by indicating to Stata where the root folder is located</li>
<li>Stata will save that path and you can then only reference sub-folders in your code</li>
</ul>
</div>
<div id="referencing-files-in-stata-1" class="slide section level1">
<h1>Referencing files in Stata</h1>
<p><strong>Exercise:</strong> create a new Stata Project in your <code>DataWork</code> folder</p>
<p><strong>1.</strong> Open Stata and then open the do-file editor.</p>
<p><strong>2.</strong> Got to <code>File &gt; New &gt; Project...</code> in the top left ribbon.</p>
<p><strong>3.</strong> Navigate to the same place where your <code>DataWork</code> folder is stored and save the project under the name “Introduction to Stata”.</p>
<p><strong>4.</strong> Right-click on your project in the Project Window and select “Add existing directory”.</p>
<p><strong>5.</strong> Select the <code>DataWork</code> folder and click “OK”</p>
</div>
<div id="referencing-files-in-stata-2" class="slide section level1">
<h1>Referencing files in Stata</h1>
<p><strong>Exercise:</strong> create a new Stata Project in your <code>DataWork</code> folder</p>
<p><strong>1.</strong> Open Stata and then open the do-file editor.</p>
<p><strong>2.</strong> Got to <code>File &gt; New &gt; Project...</code> in the top left ribbon.</p>
<p><strong>3.</strong> Navigate to the same place where your <code>DataWork</code> folder is stored and save the project under the name “Introduction to Stata”.</p>
<p><strong>4.</strong> Right-click on your project in the Project Window and select “Add existing directory”.</p>
<p><strong>5.</strong> Select the <code>DataWork</code> folder and click “OK”.</p>
<p><span style="border-left: solid 5px lightgray;padding-left: 1em;display: block;margin-block-start: 1em;margin-block-end: 1em;margin-inline-start: 40px;margin-inline-end: 40px;font-size:80%">If this worked, you will now see all the folder inside the <code>DataWork</code> folder in the Project Window.</span></p>
<p><strong>6.</strong> Right-click on your project and click “Save project”.</p>
</div>
<div id="referencing-files-in-stata-3" class="slide section level1">
<h1>Referencing files in Stata</h1>
<p><strong>Exercise:</strong> create a new Stata Project in your <code>DataWork</code> folder</p>
<p><span style="border-left: solid 5px lightgray;padding-left: 1em;display: block;margin-block-start: 1em;margin-block-end: 1em;margin-inline-start: 40px;margin-inline-end: 40px;font-size:80%">If this worked, you should see the following content in the folder for your flash drive .</span></p>
<p><img src="img/project.png" style="width:60.0%" /></p>
</div>
<div id="referencing-files-in-stata-4" class="slide section level1">
<h1>Referencing files in Stata</h1>
<p>Now that your project is set and open, you can reference files inside the <code>DataWork</code> folder by calling them through the file path that starts in <code>DataWork</code>.</p>
<ul>
<li>To open that same file, for example, you can write</li>
</ul>
<pre><code>use &quot;DataWork/Data/Final/final_process.dta&quot;</code></pre>
</div>
<div id="referencing-files-in-stata-5" class="slide section level1">
<h1>Referencing files in Stata</h1>
<p><strong>Exercise:</strong> use the <code>use</code> command in the Stata session where you have your Stata project open to load the file in <code>DataWork/Data/Clean/item_clean.dta</code></p>
</div>
<div id="referencing-files-in-stata-6" class="slide section level1">
<h1>Referencing files in Stata</h1>
<p><strong>Exercise:</strong> use the <code>use</code> command in the Stata session where you have your Stata project open to load the file in <code>DataWork/Data/Clean/item_clean.dta</code></p>
<pre><code>use &quot;DataWork/Data/Clean/item_clean.dta&quot;</code></pre>
<p><span style="border-left: solid 5px lightgray;padding-left: 1em;display: block;margin-block-start: 1em;margin-block-end: 1em;margin-inline-start: 40px;margin-inline-end: 40px;font-size:80%">Note that this will only work when you have the Stata project that you just configured open. Therefore, you should always start your work in Stata by launching this project.</span></p>
</div>
<div id="version-control" class="slide section level1">
<h1>Version control</h1>
<ul>
<li>Version control is a crucial part of data management as
<ul>
<li>Code and documents undergo a lot of revision and redrafting</li>
<li>Multiple users make changes to code and documents</li>
</ul></li>
<li>It is essential to set up version control at the start of the project to avoid glitches in later stages</li>
<li>The version control system that your team will adopt will depend on what technologies everyone is comfortable using</li>
</ul>
</div>
<div id="version-control-1" class="slide section level1">
<h1>Version control</h1>
<p><span style="color:orange"><strong>Option 1:</strong> use naming conventions</span></p>
<ul>
<li>Using file naming conventions (such as adding dates and initials as suffixes) is a very accessible option</li>
<li>It is better to do this than to have no version control</li>
<li>But it can get very unwieldy very quickly</li>
</ul>
</div>
<div id="version-control-2" class="slide section level1">
<h1>Version control</h1>
<p><img src="img/monalisa.jpeg" /></p>
</div>
<div id="version-control-3" class="slide section level1">
<h1>Version control</h1>
<p><span style="color:orange"><strong>Option 1:</strong> use naming conventions</span></p>
<ul>
<li>If you choose this option, make sure to agree on conventions for how to add names and dates</li>
<li>If you choose this option, make sure to <strong>follow</strong> these conventions</li>
</ul>
</div>
<div id="version-control-4" class="slide section level1">
<h1>Version control</h1>
<p><span style="color:orange"><strong>Option 2:</strong> syncing software</span></p>
<ul>
<li>Syncing software such as Dropbox, OneDrive and Google Drive allow teams to
<ul>
<li>Share documents</li>
<li>See who made changes to documents and when</li>
<li>Revert documents to older versions</li>
</ul></li>
<li>However, it doesn’t allow to easily keep track of specific changes made</li>
</ul>
</div>
<div id="version-control-5" class="slide section level1">
<h1>Version control</h1>
<p><span style="color:orange"><strong>Option 3:</strong> <code>git</code></span></p>
<ul>
<li><code>git</code> is a version control system designed to keep track of changes made to text files such as code</li>
<li>It allows teams to
<ul>
<li>Share documents</li>
<li>Avoid conflicts when editing documents at the same time</li>
<li>See who made <strong>which</strong> changes to documents and when</li>
<li>Revert documents to older versions</li>
</ul></li>
<li>However, it is a complex system with a steep learning curve</li>
</ul>
</div>
<div id="summary" class="slide section level1">
<h1>Summary</h1>
<ul>
<li>Discuss with your team what are the standards for
<ul>
<li>File sharing platform</li>
<li>Version control system</li>
<li>Storage and processing needs</li>
<li>Task management system</li>
<li>Where to document decisions</li>
<li>Statistical software to be used</li>
<li>Data security practices</li>
</ul></li>
<li>Organize folders in an intentional and consistent manner</li>
<li>Use Stata projects to make sure people can edit one set of codes in different computers</li>
<li>It’s fine to make adjustments as the project evolves, but having a plan before you start will avoid many hours of cleaning up your own mess</li>
</ul>
</div>
</div>
</body>
</html>
